Resource allocation optimization for multi-dimensional machine learning environments

ABSTRACT

Some embodiments of the present application include obtaining first data from a data feed to be provided to a plurality of machine learning models and detecting a changepoint in the first data. In response to the changepoint being detected, a first machine learning model may be executed on the first data to obtain first output datasets. A first performance score for the first machine learning model may be computed based on the first output datasets. A second machine learning model may be caused to execute on the first data based on the first performance score satisfying a first condition.

CROSS-REFERENCE TO RELATED APPLICATION(S)

This application is a continuation of U.S. Pat. Application No.17/328,029, filed May 24, 2021. The content of the foregoing applicationis incorporated herein in its entirety by reference.

BACKGROUND

Different machine learning models are used for different purposes (e.g.,computer vision, speech recognition, etc.). Additionally, differentmachine learning models can operate at different frequencies, with someexecuting very frequently (e.g., hourly, daily, etc.) and othersexecuting less frequently (e.g., monthly, yearly, etc.). Some machinelearning models can be configured to execute on the same data even ifthe execution frequencies of those machine learning models vary. In someinstances, the data executed on by a machine learning model can causeerrors, inconsistencies, or other issues to arise. However, these issuesare typically detected after running the machine learning model, wastingvaluable processing resources, memory, and time.

SUMMARY

Some embodiments involve optimizing resource allocation in amulti-dimensional machine learning environment or other computingenvironments. As an example, computational resource usage may be reducedvia selective execution of a machine learning model on data, where theselective execution is based on results of another machine learningmodel that executed on the data.

In some embodiments, first data from a data feed to be provided to aplurality of machine learning models may be obtained. The first data maybe analyzed to detect whether the first data includes any changepoints.In response to a changepoint being detected, a first machine learningmodel may be caused to execute on the first data to obtain first outputdatasets. A first performance score for the first machine learning modelmay be computed based on the first output datasets. A second machinelearning model may be caused to execute on the first data based on thefirst performance score satisfying a first condition.

In some embodiments, production data to be provided to a plurality ofmachine learning models may be obtained via a data feed, which may beconfigured to receive updated application data from one or morereal-time applications. The plurality of machine learning models mayinclude, for example, a first machine learning model and a secondmachine learning model, which each have a first execution frequency. Insome embodiments, a changepoint in the production data may be detectedbased on a value of a first feature of the production data beingdetermined to differ by more than a threshold amount from an expectedvalue for the first feature. In response to detecting the changepoint inthe production data, the first machine learning model and the secondmachine learning model may both be executed on the production data toobtain, respectively, first output datasets and second output datasets.In some embodiments, a first performance score for the first machinelearning model may be computed based on the first output datasets, and asecond performance score may be computed for the second machine learningmodel based on the second output datasets. In response to determiningthat the first performance score, the second performance score, or thefirst and second performance scores satisfy a condition, a third machinelearning model having a second execution frequency, less than the firstexecution frequency, may be built. The third machine learning model maybe executed on the production data. Additionally, the condition may besatisfied if the first performance score or the second performance scoreis less than a threshold performance score.

Various other aspects, features, and advantages of the invention will beapparent through the detailed description of the invention and thedrawings attached hereto. It is also to be understood that both theforegoing general description and the following detailed description areexamples and not restrictive of the scope of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a system for identifying changepoints, facilitatingexecution of prediction models, and providing feedback regarding causesof and remedies for changepoints, in accordance with one or moreembodiments.

FIG. 2 shows a process for determining whether production data includesa changepoint, in accordance with one or more embodiments.

FIG. 3 shows a graph of time series data including a detectedchangepoint, in accordance with one or more embodiments.

FIG. 4 shows a process for executing one or more machine learning modelson production data, in accordance with one or more embodiments.

FIG. 5 shows a process for computing a performance score of a machinelearning model, in accordance with one or more embodiments.

FIG. 6 shows a process for building a machine learning model, inaccordance with one or more embodiments.

FIG. 7 shows a database storing machine learning models having variousexecution frequencies, in accordance with one or more embodiments.

FIGS. 8A and 8B show flowcharts of a method for determining a machinelearning model to execute based on results of other machine learningmodels, in accordance with one or more embodiments.

FIG. 9 shows a flowchart of a method for assigning machine learningmodels to be a primary model or a secondary model, in accordance withone or more embodiments.

DETAILED DESCRIPTION

In the following description, for the purposes of explanation, numerousspecific examples are set forth in order to provide a thoroughunderstanding of example embodiments. It will be appreciated, however,by those having skill in the art that embodiments may be practicedwithout these specific details or with an equivalent arrangement.

FIG. 1 shows a system 100 for identifying changepoints, facilitatingexecution of prediction models, and providing feedback regarding causesof and remedies for changepoints, in accordance with one or moreembodiments. As shown in FIG. 1 , system 100 may include computer system102, client devices 104 a-104 n, which collectively may be referred toherein as “client devices 104” and may individually be referred toherein as “client device 104,” data feeds 140, database(s) 130, or othercomponents. Computer system 102 may include changepoint detectionsubsystem 112, model execution subsystem 114, model performancesubsystem 116, model build subsystem 118, and/or other components. Eachclient device 104 may include any type of mobile terminal, fixedterminal, or other device. By way of example, client device 104 mayinclude a desktop computer, a notebook computer, a tablet computer, asmartphone, a wearable device, or other client device. Users may, forinstance, utilize one or more client devices 104 to interact with oneanother, one or more servers, or other components of system 100. Itshould be noted that, while one or more operations are described hereinas being performed by particular components of computer system 102,those operations may, in some embodiments, be performed by othercomponents of computer system 102 or other components of system 100. Asan example, while one or more operations are described herein as beingperformed by components of computer system 102, those operations may, insome embodiments, be performed by components of client device 104. Itshould also be noted that, although some embodiments are describedherein with respect to machine learning models, other prediction models(e.g., statistical models or other analytics models) may be used in lieuof or in addition to machine learning models in other embodiments (e.g.,a statistical model replacing a machine learning model and anon-statistical model replacing a non-machine-learning model in one ormore embodiments). For instance, a machine learning model represents onetype of prediction model, however not all prediction models are requiredto be machine learning models.

In machine learning environments, different machine learning models canexecute at different execution frequencies, using the same or differentdata. For example, some machine learning models may execute on datadaily or weekly, while other machine learning models may execute on datamonthly or quarterly. In some instances, the data can cause errors withsome of the machine learning models, such as inaccurate predictions,null result sets, or other issues. However, in many cases, these issuesare not recognized until run time when the machine learning modelsexecute on the data. The outputs from the machine learning models canthen be incorrect, inconsistent, or invalid, creating technical problemssuch as valuable computational resources processing the data with themachine learning model being wasted as the model will likely need to bere-run at a later time once the data has been updated or cleaned. Insome cases, to address the issues, the model may need to be rebuilt,re-trained, or replaced with another model. This can add additionalcomplexities to the machine learning environment as the rebuilt,re-trained, or replacement model will need to be configured and thenexecuted on the data (or updated data). In addition to wastingcomputational resources, the aforementioned scenarios are timeconsuming, particularly when a model needs to be rebuilt or re-trained.In real-world applications, latency in obtaining results from a machinelearning model can be tremendously impactful.

Described herein are technical solutions to the above-mentionedtechnical problems related to non-optimal resource allocation andcomputing resource consumption, as well as machine learning resultlatency. In particular, the technical solutions described herein enableoptimized resource allocation in machine learning environments, such asthose described above. For instance, feedback from execution of somemachine learning models on data can indicate whether problems will arisewhen executing other machine learning models on that data. Inparticular, in multi-thread environments, multiple machine learningmodels may be executed in parallel or substantially in parallel. Forinstance, while one processing core is used to execute one machinelearning model, a different processing core can be used to executeanother machine learning model. When it is determined that executingsome machine learning models on the data will cause issues (e.g.,running on one or more processing cores), based on the results of othermachine learning models executing on that data (e.g., running ondifferent processing cores), preventative actions may be initiated toconserve computing resources and ensure that those models are notexecuted. For example, the models may be replaced with other existingmodels, rebuilt, or re-trained.

As another example, additional models to better handle the data and notgenerate errors may be built and deployed so as to execute on the datawithout any latency. By doing so, computing resources are preserved forthose machine learning models that will produce valid and applicableresults. Additionally, the technical solutions described herein reducelatency in obtaining valid machine learning results by minimizing anamount of time wasted on machine learning models whose results will notbe used, as well as having a model ready to execute at the desiredexecution frequency that will not cause invalid results to be produced.In some embodiments, the technical solutions may be implemented using adistributed computing environment. For example, computer system 102 mayinclude a plurality of computing devices (e.g., multiple processingcores), to implement the disclosed techniques. As a result, latency inobtaining results can be reduced from thirty hours to as few as thirtyminutes.

In some embodiments, production data may be obtained from a data feed,such as data feed 140, which may receive updated application data fromone or more real-time application. Feature sets, observed results forthe feature sets, and other information, may be extracted from theupdated application data to generate the production data. The featuresets, observed results, or other information may be extracted based onmodel input parameters of a given model or set of models with which theproduction data is to be executed on. The model input parameters mayindicate which features are relevant to a particular machine learningmodel such that the model is capable of generating output datasets. Insome embodiments, a data feed from which production data is to beobtained may be selected, and the production data may be provided as aninput to a trained machine learning model. Furthermore, data processingmay be performed on raw data obtained from data feed 140 to generate theproduction data to be used as input to the trained machine learningmodel.

In some embodiments, data feed 140 may be selected from amongst aplurality of available data feeds based on a model identifier associatedwith a trained machine learning model with which the production data isto be provided. The model identifier may indicate a machine learningmodel or a type of machine learning model stored in a model repository,such as model database 134. Alternatively or additionally, the modelidentifier may indicate a type of machine learning model that wasobtained from a training environment (e.g., a CNN for computer vision,GBM for a financial model, etc.). Based on the model identifier, datafeed 140 may be selected and updated application data may be streamedfrom the selected data feed.

Data feed 140 may be configured to receive a continuous flow of updatedapplication data from a corresponding real-time application. Forexample, a real-time application may generate and output updatedapplication data, which may be received by data feed 140. As anotherexample, the real-time application may generate and output the updatedapplication data, which may be received by another data feed.

In some embodiments, data feed 140 may be configured to receive updatedapplication data for a predetermined amount of time. For example, theupdated application data may be provided to data feed 140 within a datastream. In some cases, the updated application data may not be storedpersistently by data feed 140. In some cases, the updated applicationdata may be buffered to local cache by a computing system associatedwith the data feed (e.g., computer system 102), which may beperiodically purged to receive new updated application data from thereal-time application.

In some embodiments, data feed 140, and other components of system 100(e.g., computer system 102, database(s) 130, etc.) may form a part orall of a data pipeline. The data pipeline may include a model trainingenvironment for training a machine learning model and a model monitoringenvironment for monitoring performances of deployed machine learningmodels. The model training environment may include a data pull process,a feature engineering process, a model build process, and a modelscoring process. The data pull process may include causing training datato be retrieved from a data corpus for training a machine learningmodel. The feature engineering process may include refining the initialtraining data (and the validation data) such that the data representsfeatures needed for input to a machine learning model to be trained. Themodel build process may include training a selected machine learningmodel with the training data. The model build process may take thetraining data as inputs for the machine learning model, and may causeoutputs of the machine learning model to be fed back to the machinelearning model as input to train machine learning model (e.g., alone orin conjunction with user indications of the accuracy of outputs, labelsassociated with the inputs, or with other reference feedbackinformation). In some embodiments, the model scoring process may includetesting the accuracy of the built machine learning model to determinewhether the machine learning model has been properly trained using thevalidation data.

The model monitoring environment may be orthogonal or parallel to themodel training environment, and can enable analysis of a trained machinelearning model for production data as the production data is provided tothe trained machine learning model. The model monitoring environment mayinclude visualization processes, model stability processes, modelaccuracy processes, and alerting processes.

In some embodiments, an accuracy of a trained machine learning model maybe computed based on production data. The accuracy of the machinelearning model may indicate how well the trained machine learning isable to predict results for production data. This accuracy differs fromthe accuracy computed during the training process because the data usedto compute the accuracy during the training process is derived fromtraining data database 136. However, the accuracy of the trained machinelearning model may be determined using the production data, which isobtained from a data feed (e.g., data feed 140), and which may only beavailable for a limited amount of time (e.g., while in the data stream).

In some embodiments, the production data may be obtained from updatedapplication data, where the updated application data may be obtainedfrom data feed 140. In some embodiments, the initial production data maybe generated from the updated application data (e.g., updatedapplication data) by extracting feature sets and observed results, andthe production data may be generated by masking one or more featuresand/or results of the initial production data. As an example, theproduction data may include data items representing feature sets (e.g.,a stream of credit card applications, with each credit card applicationincluding information such as an applicant’s annual salary, residence,employment history, and the like). In some embodiments, the productiondata may include feature sets but not the corresponding observedresults. In some embodiments, however, the production data may includethe feature sets and the corresponding observed results, however theobserved results may be masked so as to not be input to the machinelearning model.

A given machine learning model may be configured to take, as input, theproduction data and generate predicted results data including resultsgenerated based on the feature sets of the production data. Thepredicted results may represent predictions from the machine learningmodel for each input feature set. In some embodiments, an accuracy scorefor the machine learning model may be computed based on the predictedresults data and the production data. For example, a difference betweenthe observed results and the predicted results may be determined, andbased on the difference and a number of feature sets included by theproduction data, an accuracy score of the machine learning model may bedetermined.

In some embodiments, a determination may be made as to whether thecomputed accuracy score satisfies a threshold accuracy condition. Forexample, the threshold accuracy condition may be satisfied if theaccuracy score is less than a threshold accuracy score. As anotherexample, the threshold accuracy condition may be satisfied if theaccuracy score is greater than or equal to a threshold accuracy score.In some embodiments, the threshold accuracy score may be determinedbased on an accuracy score previously determined for the machinelearning model during the training process. If it is determined that themachine learning model satisfies the threshold accuracy condition, thena notification may be generated indicating that the training data usedto train the machine learning model is to be updated. For example, newdata sets may be retrieved and combined with some or all of the datasets used to generate the training data. Some embodiments include usingsome of the production data, if available, to generate updated trainingdata. In some embodiments, the updated training data may be used tore-train the machine learning model, a new instance of the machinelearning model, or a new machine learning model.

In some embodiments, residuals between the predicted results and theobserved results may be computed. Residuals represent a differencebetween what is actually detected and what is predicted. For example if,for a machine learning model configured to predict a credit score for agiven credit application, a predicted credit score is 700 and an actualcredit score is 750, then the residual would be 50. In some embodiments,a graphical representation of the residuals may be generated to identifywhich feature or features contribute most or least to residuals. Forexample, the residuals may indicate that geographical location affects acredit score greater than expected or desired. In such cases, themachine learning model may, during a rebuild or subsequent training, orduring deployment, modify one or more parameters (e.g., hyperparameters)to decrease or increase the effect of geographical location on creditscore predictions. In some embodiments, an accuracy of the machinelearning model may be determined based on the residuals. For instance,because the residuals reflect the difference between the predictedresults and the observed results for a machine learning model, theaccuracy score of the machine learning model may also be determinedbased on the residuals.

In some embodiments, a stability of a model may be determined bycomputing a stability score for the machine learning model based on theproduction data and the training data. The stability score may indicatehow similar the production data being input to the machine learningmodel is to the training data used to train the machine learning model.In some embodiments, the stability score may indicate whether adistribution of features included by the production data and to be inputto the trained machine learning model is the same or similar to adistribution of features included by the training data used to train thetrained machine learning model. If the production data includes adistribution of features that greatly differs from the distribution offeatures included by the training data, then the machine learning modelmay be unable to generate predictions based on the production data, thepredictions made by the machine learning model for the production datamay be unreliable, increase latency in generating predictions for theproduction data, or cause other issues to occur. Different metrics maybe used to compute the stability score including, but not limited to(which is not to suggest that other lists are limiting), populationstability index (PSI), characteristic stability index (CSI), principalcomponent analysis (PCA), or other metrics. PSI measures an amount ofshift in a population of variables between two data sets. CSI identifieswhich variable is causing a shift in the population of variables. PCAdetermines which variable is having the greatest amount of influencefrom the population of variables. In some embodiments, a determinationmay be made as to whether the stability score satisfies a thresholdstability condition. The threshold stability condition may be satisfiedif the computed stability score for the production data is less than athreshold stability score. As an example, a determination may be made asto whether a PSI value, determined based on the training data and theproduction data, is less than a threshold PSI value. If so, the PSIvalue (e.g., the stability score) may be classified as satisfying thethreshold stability condition. Some embodiments include generating anotification to update the training data and/or cause the trainedmachine learning model to be rebuilt in response to determining that thecomputed stability score (e.g., the PSI value) satisfies the thresholdstability condition (e.g., the PSI value is less than the threshold PSIvalue).

In some embodiments, a determination may be made as to whether aparticular value of a feature included within the production datadiffers from an expected value for that feature based on the trainingdata. For example, if the training data used to train the machinelearning model included credit card applications, each credit cardapplication may include a feature of annual salary for the applicant. Avalue provided by each application for this feature may be extractedfrom each application, and an average value for this feature may bedetermined. For example, the average annual salary of applicantsincluded within credit card applications used to train the machinelearning model to approve/not-approve each applicant for a credit cardmay be computed by summing the value of annual salary from each creditcard application and dividing the summed value by the number ofapplications to obtain the average value. When the production data isanalyzed, a determination can be made as to whether a value associatedwith the annual salary feature for a given data item of the productiondata differs from the average value by more than a threshold amount. Ifso, then this may indicate that some of the production data does notreflect the data expected to be input to the trained machine learningmodel. Therefore, a notification may be generated to cause the trainingdata to be updated and/or the machine learning model to be rebuilt. Insome embodiments, a determination is made as to whether a number ofinstances of a value associated with a given feature in the productiondata differing from an expected value for that feature occurs in theproduction data more than a threshold number of times. For example, asingle instance of a value for average salary exceeding the thresholdamount may not necessitate updating the training data or rebuilding themodel.

In some embodiments, a similarity score between the training data andthe production data may be determined based on the expected pattern offeatures represented by the training data and observed patterns offeatures represented within the production data. For example, theexpected pattern of features represented by the training data mayindicate a substantially static distribution of features included by thetraining data. It may then be determined whether the patterns offeatures observed within the production data is also the same or similarto the pattern of features within the training data. In someembodiments, a determination is made as to whether a thresholdsimilarity condition is satisfied (e.g., if the similarity score isgreater than or equal to a threshold similarity score) indicating howsimilar the training data and the production data are. In response todetermining that the similarity score fails to satisfy the thresholdsimilarity condition, the machine learning model may be rebuilt and/orthe training data used to train the machine learning model may beupdated.

In some embodiments, a number of anomalies within the production datamay be detected. For example, a number of NULL values present within theproduction data may be determined. A NULL value, which may also bereferred to herein interchangeably as a “NULL entry,” indicates that adata value for a given data field does not exist. For example, a NULLvalue for the feature “annual salary,” (e.g., one type of data fieldthat data can be input to in an example credit card application), mayindicate that no value exists for this feature for a particular dataitem. A determination may be made as to whether a number of NULL valuesin the production data is greater than a threshold number of NULL valuesand, if so, a notification may be generated to cause the training datato be updated and/or the machine learning model to be rebuilt.

Additional details related to the data pipeline, training environment,and model monitoring environment, are included in U.S. Pat. ApplicationNo. 17/089,645, filed on Nov. 4, 2020, the disclosure of which isincorporated herein by reference in its entirety.

Production data may be generated from the updated application dataretrieved from a selected data feed. In some embodiments, updatedapplication data may include a plurality of data items representing aplurality of feature sets and observed results respectivelycorresponding to each of the plurality of feature sets. Each of thefeature sets may include one or more types of features represented bythe updated application data. For example, one of feature sets mayinclude the feature type “salary information,” corresponding to afeature “salary,” which may be used as a model input parameter to afinancially-related prediction model. Each of the observed results mayindicate a result obtained via an automated decision process, anenhanced review decision process, or other decision making process. Insome embodiments, the automated decision process may be made inreal-time. The automated decision process may provide a result given aninput data item having one or more feature sets within a predeterminedamount of time from the data item being input. For example, the observedresult may be obtained from the automated decision process in less than30 seconds, less than 15 seconds, less than 10 seconds, less than 5seconds, or within other amounts of time. In some embodiments, theautomated decision process may determine an observed result for a dataitem based on the feature sets, and thus the features, represented bythe data item. Furthermore, different feature sets can cause differentobserved results.

As an example, the automated decision process may correspond to areal-time credit card application approval process. The examplereal-time credit card application approval process may take the inputfeatures provided by an individual and determine whether the individualis approved/not approved for a credit card based on the input features.For example, an individual may input annual salary information, lengthof employment, geographical information, and the like, into thereal-time credit card application. Based on these inputs, the real-timecredit card application approval process may either approve or notapprove the individual for the credit card. The approval/non-approval ofthe individual may represent an example of an observed result, where thecredit card application may represent the data item, and the informationprovided by the individual via the credit card application may representthe feature sets including features such as annual salary information,length of employment, geographical information, etc.

As another example, the automated decision process may correspond to areal-time credit determination process, whereby an amount of credit isdetermined for an individual based on information provided by a user toa credit application. Similar to the real-time credit card applicationapproval process example described above, an individual may provide, asan input, annual salary information, length of employment, geographicalinformation, and the like, into the real-time credit determinationprocess. Based on the inputs, the real-time credit determination processmay determine an amount of credit to allocate to the individual (or anaccount associated with the individual). The amount of credit mayrepresent another example of an observed result, where the creditapplication may represent the data item, and the information provided bythe individual via the credit application may represent the feature setsincluding features such as annual salary information, length ofemployment, geographical information, etc.

While the aforementioned examples relate to financial real-timeautomated processes, applications outside of finance are alsoapplicable. For example, the automated decision process may correspondto an autonomous driving decision process. In this example, thereal-time object detection process may take, as an input, an image, aframe from a real-time video feed, a feature vector indicating objectsdetected within an image or frame from a video, and the like. Based onthe inputs, the autonomous driving decision process may generate adecision regarding a driving maneuver to be performed by a vehicle. Forexample, the decision may include causing the vehicle to turn right orleft, how much to turn the vehicle, whether to speed up or slow down thevehicle, or to perform a different maneuver. The maneuver to beperformed may represent yet another example of an observed result, wherethe input image, frame, feature vector, etc. may represent the dataitem, and detected objects may represent the feature sets includingfeatures such as type of object detected, distance to the detectedobject, velocity, directionality, etc.

As an example, the enhanced review decision process may correspond to asubsequent review of the credit card application approval process. Forinstance, after the real-time credit card application approval processgenerates an initial result, the credit card application and initialresult may be provided to an enhanced review system for determiningwhether the initial result was correct. This may include providing thecredit card application to a robust set of algorithms to determine theeligibility of the individual, allowing one or more human reviewers toanalyze the credit card application, and the like. Based on the enhancedreview system’s analysis of the real-time credit card applicationapproval process, an approval/non-approval of the individual’s creditcard application may be generated. In this example, the observed resultmay represent the approval/non-approval of the individual’s credit cardapplication, where the credit card application may represent the dataitem, and the information provided by the individual via the credit cardapplication may represent the feature sets including features such asannual salary information, length of employment, geographicalinformation, etc.

As another example, the enhanced review decision process may correspondto a subsequent review of the credit determination process. In thisexample, an enhanced review system may determine whether the real-timecredit determination process generated an appropriate amount of creditfor an individual based on information provided by the individual via acredit application. This may include providing the credit cardapplication to a robust set of algorithms to determine an amount ofcredit to be allotted to the individual, allowing one or more humanreviewers to analyze the credit card application and determine an amountof credit to be allotted to the individual, and the like. Based on theenhanced review system’s analysis of the credit determination process,an amount of credit to be allocated to the individual (or an accountassociated with the individual) may be generated. Similar to thereal-time credit determination process, the amount of credit determinedby the enhanced review system’s analysis may represent another exampleof an observed result, where the credit application may represent thedata item, and the information provided by the individual via the creditapplication may represent the feature sets including features such asannual salary information, length of employment, geographicalinformation, etc.

As yet another example, the enhanced review decision process maycorrespond to a subsequent review of the autonomous driving decisionprocess. In this example, an enhanced review system may determinewhether the autonomous driving decision process generated an appropriatedecision regarding a driving maneuver to be performed by a vehicle. Thismay include providing the input information (e.g., the image, frame fromthe video feed, feature vector, etc.) to a robust set of algorithms todetermine a maneuver to be performed, allowing one or more humanreviewers to analyze the input information and determine a maneuver tobe performed, and the like. Based on the enhanced review system’sanalysis of the autonomous driving decision process, a maneuver to beperformed may be generated. For example, the decision may includecausing the vehicle to turn right or left, how much to turn the vehicle,whether to speed up or slow down the vehicle, or to perform a differentmaneuver. In some embodiments, the maneuver determined by the enhancedreview system may differ from the real-time autonomous driving decisionprocess’s result. The maneuver to be performed, determined by theenhanced review system, may represent yet another example of an observedresult, where the input image, frame, feature vector, etc., mayrepresent the data item, and detected objects may represent the featuresets including features such as type of object detected, distance to thedetected object, velocity, directionality, etc.

In some embodiments, the production data may be monitored for detectionof a changepoint. A changepoint represents instances of data abruptlyshifting in some manner. In particular, changepoints represent abruptshifts in time series data (e.g., data that is sequential in time). Agoal of changepoint detection is to identify a location (e.g., a time)that a particular changepoint or changepoints occurred in the data, aswell as determining a number of changepoints in the data. Productiondata, P_(t), as described herein, is one example of time series data:

Production Data : P_(t) = (P₁, P₂, ⋯, P_(n))

In Equation 1, each data point P_(t), with t = 1, 2, ..., n, representsan observed value of the data at time t. When a changepoint is detectedin the production data, the data experiences some sort of abrupt anddistinct change. Some example changes that can represent changepointsinclude, but are not limited to (not to suggest that other lists arelimiting), mean shifts (e.g., a shift in the mean value of the data) ora slope change.

In some embodiments, the production data may be analyzed to identify aset of candidate changepoints. In particular, in a multiple changepointdetection environment, which can occur when analyzing real world data,the total number of different multiple changepoints may be of the order2^(n). Therefore, candidate changepoints (e.g., multiple changepoints)may be referred to as models. For each model (e.g., candidate multiplechangepoint), a changepoint score can be computed. For example, aBayesian Minimum Description Length (BMDL) score may be computed. Aftercomputing all of the BMDL scores, a model having a smallest BMDL scoremay be selected, and all changepoints included in that model can beclassified as detected changepoints.

In some embodiments, a magnitude of the changepoint detected may becomputed to determine whether the input data stream from data feed 140has become corrupted. For example, a slope and mean before thechangepoint and a slope and mean after the changepoint may be comparedto determine if the change in the slope and the mean exceeds a thresholdslope change and/or a threshold mean change. If the change in the slopeor mean is greater than the threshold slope or mean change, then thismay indicate that a new machine learning model needs to be built or acurrent machine learning model needs to be rebuilt or re-trained.

If a changepoint is detected in the production data, then one or moremachine learning models may be executed on the production data. Forexample, after detecting a changepoint in the production data, theproduction data may be provided to a first machine learning model and asecond machine learning model having a same or similar executionfrequency. Additionally, the production data may be provided to othermachine learning models having different execution frequencies. Theexecution frequency represents a cadence with which a particular machinelearning model executes. For example, a machine learning model may havea daily execution frequency (e.g., executes every day), a weeklyexecution frequency (e.g., executes every week), a monthly executionfrequency (e.g., executes every month/30 days), or other executionfrequencies. In some embodiments, the first and second machine learningmodels may have a first execution frequency (e.g., daily, weekly, etc.).Other machine learning models, which are also to be provided with theproduction data, may have a second execution frequency (e.g., monthly,quarterly, etc.).

In response to detecting the changepoint in the production data, thefirst and second machine learning models may be executed on theproduction data, thereby obtaining first output datasets and secondoutput datasets, respectively. The first and second output datasets maybe used to compute a performance score for the first and second machinelearning models. For example, a mean, variance, autocovariance,quantile, or other metrics, may be computed for the first and secondmachine learning models. Alternatively, other metrics indicating aquality of a machine learning model may be computed using the outputdatasets. For example, a distribution of the output datasets may becomputed.

In some embodiments, a determination may be made as to whether theperformance scores computed for each of the first and second machinelearning models satisfy a condition. The condition, for example, may besatisfied when a performance score is less than or equal to a thresholdperformance score. The threshold performance may be determined based onhistorical performance scores for a given machine learning model. Forexample, the performance score may be computed by determining howaccurately a given machine learning model predicted a particular resultbased on training data used to train that machine learning model ascompared to the result obtained via the machine learning model executingon the production data. The closer the predicted result is to the actualresult obtained, the better the performance of the model is said to be.

Models that poorly predict results may require certain actions to betaken. In some embodiments, if either the first or second machinelearning model has a performance score that satisfies the condition(e.g., the performance score is less than or equal to a thresholdperformance score), then this may indicate that a corresponding modelmay need to be re-trained or rebuilt. Retraining a model may includeupdating the training data used to train the model.

In some embodiments, as mentioned above, the production data is to beprovided to other machine learning models having different executionfrequencies than that of the first and second machine learning models.Given that the performance of the first or second machine learningmodels, when executed on the production data, failed to satisfy thecondition, the other machine learning models may experience problems aswell. However, because some of the other machine learning models run ata different execution frequency, these problems may not arise untilafter those machine learning models execute on the production data. Toprevent such problems, and thus wasting computing resources, in responseto determining that either the first or second machine learning model’scorresponding performance score does not satisfy the condition, theproduction data may be provided to a different machine learning modelthan originally intended. For instance, in some embodiments, a thirdmachine learning model, having a second execution frequency differentthan the first execution frequency of the first and second machinelearning models may be selected to execute on the production data. As anexample, the selected machine learning model may be better configured tohandle the production data than the machine learning model originallyintended to be executed on the production data, thereby optimizingresource allocation to a model that will produce valid and usefulresults.

In some embodiments, the third machine learning model may be built inresponse to the determination that the first or second machine learningmodels’ performance score does not satisfy the condition. For example,the third machine learning model may be built such that the modelparameters of the third machine learning model are not dependent on thefeatures in the production data with which the changepoint (orchangepoints) are associated. For example, if a changepoint in theproduction data is detected for a particular variable, such as a creditscore, grayscale, or other feature, then the third machine learningmodel may be configured such that that particular variable has minimalor no impact on the output datasets of the model. Therefore, the machinelearning environment’s computing resources can be conserved for use withmodels that will produce valid and useful results, and can also minimizethe amount of computing resources being used by models that could havedifficulties handling the production data (e.g., due to the variable inthe production data having the changepoint and the model parameters ofthat model). Consequently, latency in obtaining results is minimized,thereby improving operating efficiency of the machine learningenvironment.

Subsystems 112-118

In some embodiments, changepoint detection subsystem 112 is configuredto detect instances of changepoints in data. For instance, changepointdetection subsystem 112 may determine whether production data to beexecuted on by one or more machine learning models includes anychangepoints. Changepoint detection subsystem 112 may retrieveproduction data from data feed 140. Data feed 140 may receive updatedapplication data from a real-time application. In some embodiments, theproduction data may be stored in production data database 132, andchangepoint detection subsystem 112 may retrieve the production datafrom production data database 132 instead. Additionally, a single datafeed (e.g., data feed 140) is depicted in FIG. 1 for illustrativepurposes only, and system 100 may include additional data feeds.

Data feed 140 may be configured to receive a continuous flow of updatedapplication data from a corresponding real-time application. Forexample, a real-time application may generate and output updatedapplication data, which may be received by data feed 140. Data feed 140may be configured to receive updated application data for apredetermined amount of time. For example, the updated application datamay be provided to data feed 140 within a data stream. In some cases,the updated application data may not be stored persistently by data feed140. In some cases, the updated application data may be buffered tolocal cache by a computing system associated with data feed 140 (e.g.,computer system 102), which may be periodically purged to receive newupdated application data from the real-time application.

Production data may be generated from the updated application data. Insome embodiments, the updated application data may include a pluralityof data items representing a plurality of feature sets and observedresults respectively corresponding to each of the plurality of featuresets. Each of the feature sets may include one or more types of featuresrepresented by the updated application data. For example, one of thefeature sets may include the feature type “salary information,”corresponding to a feature “salary,” which may be used as a model inputparameter to a financially-related prediction model.

Each observed result may indicate a result obtained via an automateddecision process, an enhanced review decision process, or other decisionmaking process. In some embodiments, the automated decision process maybe made in real-time. The automated decision process may provide aresult given an input data item having one or more feature sets within apredetermined amount of time of the data item being input. For example,the observed result may be obtained from the automated decision processin less than 30 seconds, less than 15 seconds, less than 10 seconds,less than 5 seconds, or within other amounts of time. In someembodiments, the automated decision process may determine an observedresult for a data item based on the feature sets, and thus the features,represented by the data item. Furthermore, different feature sets cancause different observed results.

As an example, the automated decision process may correspond to areal-time credit card application approval process. The examplereal-time credit card application approval process may take the inputfeatures provided by an individual and determine whether the individualis approved/not approved for a credit card based on the input features.For example, an individual may input annual salary information, lengthof employment, geographical information, and the like, into thereal-time credit card application. Based on these inputs, the real-timecredit card application approval process may either approve or notapprove the individual for the credit card. The approval/non-approval ofthe individual may represent an example of an observed result, where thecredit card application may represent the data item, and the informationprovided by the individual via the credit card application may representthe feature sets including features such as annual salary information,length of employment, geographical information, etc.

As another example, the automated decision process may correspond to anautonomous driving decision process. In this example, the real-timeobject detection process may take, as an input, an image, a frame from areal-time video feed, a feature vector indicating objects detectedwithin an image or frame from a video, and the like. Based on theinputs, the autonomous driving decision process may generate a decisionregarding a driving maneuver to be performed by a vehicle. For example,the decision may include causing the vehicle to turn right or left, howmuch to turn the vehicle, whether to speed up or slow down the vehicle,or to perform a different maneuver. The maneuver to be performed mayrepresent yet another example of an observed result, where the inputimage, frame, feature vector, etc. may represent the data item, anddetected objects may represent the feature sets including features suchas type of object detected, distance to the detected object, velocity,directionality, etc.

The resulting production data produced by data feed 140, andsubsequently retrieved or streamed to changepoint detection subsystem112, may therefore represent time series data, such as P_(t) = (P₁, P₂,•••, P_(n)) described above with respect to Equation 1. In someembodiments, the time series data may be multi-dimensional. For example,the time series data may include multiple variables (e.g., a price of anitem, a salary, a credit score, a grayscale value, etc.), some of whichmay be independent variables, while others may be correlated. Thus,changepoints, which represent abrupt changes in a value of the data, mayoccur in one or more dimensions. In some embodiments, each dimension isorthogonal to the other dimensions. For example, consider ann-dimensional feature vector residing in an n-dimensional feature space.Each feature is representative of one dimension in the n-dimensionalfeature space. Thus, each variable represented in the data resides in adifferent dimension of an n-dimensional feature space which, so long asn is greater than 1, corresponds to a multi-dimensional space.

Changepoint detection subsystem 112 may be configured to detect multiplechangepoints in time (e.g., time t = {1, 2, ... , n}) for the productiondata. For example, with reference to FIG. 2 , process 200 includesproduction data 202 being provided to changepoint detection subsystem112. Production data 202 may be obtained from data feed 140, productiondata database 132, or another data source. Changepoint detectionsubsystem 112 may be configured to detect candidate multiple changepointconfigurations in production data 202, which may each be referred to asa model or candidate model. Changepoint detection subsystem 112 canperform model selection based on the Bayesian Minimum Description Length(BDML) framework. Some embodiments include computing a BDML score foreach candidate model and selecting the candidate model having thesmallest BDML score. All changepoints included in the selected candidatemodel may be classified as detected changepoints.

As mentioned above, changepoints represent abrupt changes in time seriesdata (e.g., in production data 202). Some examples of changes that canoccur are shifts in a mean value of the data and/or shifts in a slope ofthe data. Production data 202, therefore, can include different regimeswhich will have one mean and slope value before a given changepoint anda different mean and slope value after the changepoint. As an example,with reference to FIG. 3 , graph 300 includes a single changepoint 302.Graph 300 depicts an example of an average hourly pay (e.g., in Americandollars) over a given temporal range. Each data point corresponds to acertain value of the average hourly pay for a particular time. In someembodiments, graph 300 may be fitted using a model for determiningwhether the data includes a changepoint (or multiple changepoints).After fitting, the model can be segmented into regions corresponding todata before the changepoint and after the changepoint. For example,graph 300 includes two regimes, first regime 310 occurring beforechangepoint 302, and second regime 320 occurring after changepoint 302.Both the intercept and slope of the production data represented in graph300 differ between first regime 310 and second regime 320.

Given a candidate changepoint model, an observed data value P_(t) attime t can be represented by Equation 2:

$\begin{array}{l}{P_{t} = \alpha_{1} + \beta_{1}\mspace{6mu} t + \alpha_{r{(t)}} +} \\{\beta_{r{(t)}}\mspace{6mu} t\left( {\sum_{i = 1}^{k}\left\lbrack {\theta_{i,1} \cdot sin\mspace{6mu} sin\left( \frac{2\pi ti}{T} \right) + \theta_{i,2} \cdot cos\mspace{6mu} cos\left( \frac{2\pi ti}{T} \right)} \right\rbrack} \right) + \varepsilon_{t}}\end{array}$

In Equation 2, the first term represents the linear segment in Regime 1,the second term represents the linear segment in Regime r, the thirdterm is a harmonic function representing a seasonal mean cycle, and thefourth term represents autocorrelated error. Although the values of theintercept and slope, α and β, respectively, are not known, they dodiffer between the first and second regimes, as illustrated by firstregime 310 and second regime 320 of graph 300. The harmonic function isto account seasonal-related fluctuations, such as, for example, the costof a particular item during different times of year.

Changepoint detection subsystem 112 may be configured to apply Equation2 to the production data to detect instances of changepoints. Additionaldetails regarding changepoint detection techniques, such as thoserelated to error correction and derivation of the BDML expression can befound in “Automating Data Monitoring: Detecting Structural Breaks inTime Series Data Using Bayesian Minimum Description Length,” Li et al.,2019; and “Multiple Changepoint Detection with Partial Information onChangepoint Times,” Li et al., 2019, which are each incorporated hereinby reference in their entirety. In some embodiments, changepointdetection subsystem 112 receives production data 202 as well as expectedfeature values 204. Using production data 202 and expected featurevalues 204, changepoint detection subsystem 112 can determine whetherproduction data 202 includes one or more changepoints. For example,expected feature values may represent a predetermined or dynamicallycomputed value for a given feature. Returning to FIG. 3 , an expectedvalue for the feature of average hourly pay shortly almost midwaythrough 2013 is approximately 31.50. However, as can be seen in graph300, there is an abrupt change from the expected value of the averagehourly pay midway through 2013, whereby the average hourly pay increasesdramatically.

Changepoint detection subsystem 112 may generate an indicator reflectingwhether any changepoints were detected. For example, changepointdetection subsystem 112 may generate a changepoint indicator 206indicating that at least one changepoint was detected in production data202. In some embodiments, changepoint indicator 206 includes datarepresenting a number of changepoints detected in production data 202, alocation (temporally) of each changepoint in production data 202, anorder (e.g., first changepoint, second changepoint, etc.) of thedetected changepoint(s), or other information. For example, changepointindicator 206 may be a tuple of {Changepoint, Time}, however otherformats can be used. If no changepoints are detected in production data202, changepoint detection subsystem 112 may generate and outputno-changepoint indicator 208. No-changepoint indicator 208 may indicatefeatures regarding production data 202, such as metrics computed forproduction data 202, as well as a NULL flag indicating a lack ofdetected changepoints. In some embodiments, changepoint indicator 206and no-changepoint indicator 208 may include data indicating parametervalues for α and β, error correction values, or other data. This canallow for additional insight to be made with respect to the behavior ofproduction data 202, which can be used for future modeling, analysis,and/or training.

In some embodiments, changepoint detection subsystem 112 is configuredto generate alerts, notifications, messages, or other communicationsindicating that a changepoint has been detected. For example, upondetecting a changepoint in production data 202, changepoint detectionsubsystem 112 may generate a message including changepoint indicator206, which may be provided to a user, such as a system administrator,via that user’s corresponding client device 104. In some embodiments,the message may also indicate actionable recommendations, such aswhether the alert is to be escalated, a model is to be rebuilt, or otherinformation.

In some embodiments, model execution subsystem 114 is configured toexecute one or more of a plurality of machine learning models stored inmodel database 134. As mentioned previously, some of the machinelearning models may have a first execution frequency (e.g., daily,weekly, etc.), while others may have a second, different, executionfrequency (e.g., monthly, quarterly, etc.). Furthermore, some of themachine learning models stored in model database 134 take, as input,different model parameters to obtain different results. For example,some machine learning models may take as model inputs credit score,annual salary, years of employment, etc., while others may take as modelinputs color gradients, edge locations, pixel locations of landmarks,etc. The model input parameters may indicate which features are relevantto a particular machine learning model such that the model is capable ofgenerating output datasets. The machine learning models may be selectedfrom model database 134 based on a particular process, task, orobjective sought to be obtained by the model. For example, aconvolutional neural network (CNN) may be selected from model database134 for processes related to computer vision. The various machinelearning models stored by model database 134, from which model executionsubsystem 114 may select from, include, but are not limited to (which isnot to suggest that any other list is limiting), any of the following:Ordinary Least Squares Regression (OLSR), Linear Regression, LogisticRegression, Stepwise Regression, Multivariate Adaptive RegressionSplines (MARS), Locally Estimated Scatterplot Smoothing (LOESS),Instance-based Algorithms, k-Nearest Neighbor (KNN), Learning VectorQuantization (LVQ), Self-Organizing Map (SOM), Locally Weighted Learning(LWL), Regularization Algorithms, Ridge Regression, Least AbsoluteShrinkage and Selection Operator (LASSO), Elastic Net, Least-AngleRegression (LARS), Decision Tree Algorithms, Classification andRegression Tree (CART), Iterative Dichotomizer 3 (ID3), C4.5 and C5.0(different versions of a powerful approach), Chi-squared AutomaticInteraction Detection (CHAID), Decision Stump, M5, Conditional DecisionTrees, Naive Bayes, Gaussian Naive Bayes, Causality Networks (CN),Multinomial Naive Bayes, Averaged One-Dependence Estimators (AODE),Bayesian Belief Network (BBN), Bayesian Network (BN), k-Means,k-Medians, K-cluster, Expectation Maximization (EM), HierarchicalClustering, Association Rule Learning Algorithms, A-priori algorithm,Eclat algorithm, Artificial Neural Network Algorithms, Perceptron,Back-Propagation, Hopfield Network, Radial Basis Function Network(RBFN), Deep Learning Algorithms, Deep Boltzmann Machine (DBM), DeepBelief Networks (DBN), Convolutional Neural Network (CNN), Deep MetricLearning, Stacked Auto-Encoders, Dimensionality Reduction Algorithms,Principal Component Analysis (PCA), Principal Component Regression(PCR), Partial Least Squares Regression (PLSR), Collaborative Filtering(CF), Latent Affinity Matching (LAM), Cerebri Value Computation (CVC),Multidimensional Scaling (MDS), Projection Pursuit, Linear DiscriminantAnalysis (LDA), Mixture Discriminant Analysis (MDA), QuadraticDiscriminant Analysis (QDA), Flexible Discriminant Analysis (FDA),Ensemble Algorithms, Boosting, Bootstrapped Aggregation (Bagging),AdaBoost, Stacked Generalization (blending), Gradient Boosting Machines(GBM), Gradient Boosted Regression Trees (GBRT), Random Forest,Computational intelligence (evolutionary algorithms, etc.), ComputerVision (CV), Natural Language Processing (NLP), Recommender Systems,Reinforcement Learning, Graphical Models, or separable convolutions(e.g., depth-separable convolutions, spatial separable convolutions,etc.).

In some embodiments, model execution subsystem 114 obtains productiondata 202, an indication of a set of machine learning models thatproduction data 202 is to be executed on, and an indication of whetherproduction data 202 includes any detected changepoints. Model executionsubsystem 114 can execute, or facilitate execution of, some or all ofthe selected machine learning models from the set of machine learningmodels. As an example, with reference to FIG. 4 , model executionsubsystem 114 may receive production data 202, an indication from modeldatabase 134 of one or more machine learning models that production data202 is to be executed on, and changepoint indicator 206 orno-changepoint indicator 208. In some cases, model database 134 mayprovide the machine learning models to be executed in addition to, orinstead of, the indication of the machine learning models.

Model execution subsystem 114 may include a set of modules, including atimer 402, a model selector 404, data duplication 406, data distribution408, other modules, or other components. Each module of model executionsubsystem 114 may be implemented by one or more processors executingcomputer program instructions stored in memory of computer system 102.

In some embodiments, timer 402 is configured to track an amount of timethat has elapsed since a machine learning model has executed, an amountof time that has elapsed since production data has been retrieved, orother time periods. As mentioned previously, machine learning models mayhave various execution frequencies with which each runs. For example,one machine learning model may execute weekly, while another machinelearning model may execute monthly. When both of these machine learningmodels are deployed to a production environment, timer 402 can determinewhether a machine learning model is to run and/or when the machinelearning model is to run. In some embodiments, timer 402 may be aphysical timer having hardware components configured to monitor anamount of time that has elapsed since a particular event (e.g., aspring-based timing mechanism, a quartz clock, etc.), computer software(e.g., an electronic oscillator), or another timing mechanism. Whentimer 402 determines that a predetermined amount of time has elapsedcorresponding to an execution frequency of a machine learning model,timer 402 may be configured to generate a trigger to cause one or moreactions facilitating a machine learning model’s execution. For example,if the execution frequency of a machine learning model is weekly, timer402 may determine when seven days has elapsed since the machine learningmodel executed.

Model selector 404 may be configured to select a particular machinelearning model to be run. For instance, model selector 404 may select amachine learning model from model database 134 based on the indicationreceived by model execution subsystem 114, the trigger generated bytimer 402, or other aspects. Model selector 404 can identify a modelidentifier of the models to executed from model database 134. Afteridentifying the model identifier, model selector 404 may select themachine learning model from model database 134 or locate the machinelearning model in the production environment such that production data202 can be provided to the appropriate models.

In some embodiments, two or more machine learning models may be executedon the same production data at the same time. For example, two machinelearning models each having a same execution frequency may execute onproduction data 202. In some cases, data duplication 406 may generateduplicates of the data included in production data 202, and datadistribution 408 can distribute each instance of production data 202 toa corresponding machine learning model. For example, machine learningmodels 410 a and 410 b may each have a first execution frequency. Eachinstance of production data 202 may be provided to machine learningmodels 410 a and 410 b by data distribution 408 such that machinelearning models 410 a and 410 b can execute on production data 202.

In some embodiments, model execution subsystem 114 is configured toexecute certain machine learning models at different times based onwhether a changepoint has been detected in production data 202. Forexample, upon receiving changepoint indicator 206, model executionsubsystem 114 may cause machine learning model 410 a and machinelearning model 410 b, each having a first execution frequency, to beexecuted on production data 202. Machine learning models 410 a and 410 bmay generate output datasets 412 a and 412 b, respectively, based onproduction data 202. As another example, upon receiving no-changepointindicator 208, model execution subsystem 114 may cause machine learningmodel 410 a to execute on production data 202 instead of machinelearning model 410 b.

As mentioned above, model execution subsystem 114 may cause differentmachine learning models to execute at different times based on theexecution frequency of each machine learning model. In some embodiments,upon timer 402 determining that a first amount of time associated with afirst execution frequency of machine learning models 410 a and 410 b haselapsed, model execution subsystem 114 may cause machine learning models410 a and 410 b to execute on production data 202. However, machinelearning model 410 n (and/or other machine learning models) may notexecute because machine learning model 410 n has a different executionfrequency than machine learning models 410 a and 410 b. For example,machine learning models 410 a and 410 b may have a weekly executionfrequency, whereas machine learning model 410 n may have a monthlyexecution frequency. In response to timer 402 determining that a secondamount of time associated with a second execution frequency of machinelearning model 410 n has elapsed, model execution subsystem 114 maycause machine learning model 410 n to execute on production data 202 tocause datasets 412 n to be generated.

In some embodiments, model performance subsystem 116 may be configuredto compute a performance score for each machine learning model. Theperformance score may indicate how well a given machine learning modelperformed. In some embodiments, the performance score may be compared toprior performance scores of the machine learning model to determinewhether the model’s performance has improved, stayed the same, orworsened. If the performance of the machine learning model has decreasedover time, then this may indicate a need to re-train, rebuild, orreplace that machine learning model.

As an example, with reference to FIG. 5 , model performance subsystem116 may obtain datasets 412 (e.g., one or more of datasets 412 a-412 n)from model execution subsystem 114. Datasets 412 may be provided tomodel performance subsystem 116 upon generation by a given machinelearning model. In some cases datasets may be stored in local cacheuntil analysis is to be performed. Model performance subsystem 116 mayuse datasets 412 to determine how well a given machine learning modelperformed based on historical performance data representing pastperformance results of that machine learning model (e.g., for previousiterations). Based on the performance of the machine learning model,model performance subsystem 116 may determine any actions to be taken(e.g., retraining, rebuilding, replacement, etc.).

In some embodiments, performance metric 502 may be selected based on thetype of machine learning model that produced datasets 412. For example,if machine learning model 410 a is a recurrent neural network (RNN),then performance metric 502 may be a residuals metric. Performancemetric 502 may be included in a set of performance metrics selected inadvance such that any datasets produced by a machine learning modelduring deployment are to have those performance metrics computed. Thevarious types of performance metrics include, but are not limited to(which is not to suggest that other listings are limiting), residuals,variance, bias, or other metrics. Upon receiving datasets 412, modelperformance subsystem 116 may compute a performance metric 502. In somecases, multiple performance metrics may be computed in parallel orsequentially for datasets 412 using a multi-thread computingenvironment. Furthermore, one or more performance metrics may beperformed in parallel or sequentially for multiple instances of datasets412 using the multi-thread computing environment. For example, residualsmay be computed for datasets 412 a and 412 b, produced by machinelearning models 410 a and 410 b, respectively, where the residuals maybe computed using separate computing threads of the multi-threadcomputing environment. A performance score 504 may be obtain uponcomputing performance metric 502. Performance score 504 may be anumerical value (e.g., a number between 0-100, a number between 0-1,etc.), a percentage, or other representation. Each performance score 504may be stored in performance database 138 for subsequent machinelearning model analysis. In some embodiments, upon generation,performance score 504, a timestamp of (i) when the performance score wasgenerated, (ii) datasets 412 were generated, and/or (iii) when acorresponding machine learning model executed, an indication of themachine learning model, a type of machine learning model, or otherinformation associated with datasets 412, may be provided to performancedatabase 138 for storage. Performance database 138 may store eachperformance score 504 in a data structure associated with thecorresponding machine learning model that produced datasets 412 so thatfuture analysis of performance scores can be easily retrieved.

In some embodiments, model performance subsystem 116 may retrieve aprior performance score 506 for a machine learning model, which can beused to determine whether the machine learning model’s performance hasimproved, stay consistent, or degraded in a current iteration. Forexample, prior performance score 506 may represent a performance scorecomputed for machine learning model 410 a prior to a most recenttraining and deployment of machine learning model 410 a. In response toobtaining datasets 412 a from machine learning model 410 a, output basedon machine learning model 410 a being executed on production data 202during a current deployment cycle, model performance subsystem 116 mayaccess performance database 138 and retrieve prior performance score 506reflecting a previously computed performance score of machine learningmodel 410 a during a previous deployment cycle. In some embodiments,prior performance score 506 may be an aggregation of multiple priorperformance scores for a corresponding machine learning model. Forexample, prior performance score 506 may be an average of performancescores computed for a machine learning model for each previousdeployment cycle.

Performance score 504 and prior performance score 506 may be compared todetermine a performance difference 508. Performance difference 508 mayindicate a change in performance of a machine learning model from onedeployment cycle to another. For example, during a previous deploymentcycle, a machine learning model may have a first performance score S1(e.g. prior performance score 506). During a current deployment cycle,the machine learning model may have a second performance score S2 (e.g.,performance score 504). Performance difference 508 may indicate adifference between first performance score S1 and second performancescore S2 (e.g., S2-S1). While the aforementioned example uses aperformance “difference,” it should be understood by those of ordinaryskill in the art that other comparison techniques may be used todetermine how a current performance score compares to a previousperformance score or scores.

In some embodiments, model performance subsystem 116 may determinewhether performance difference 508 satisfies one or more conditions 510.As an example, one of conditions 510 may be satisfied if performancedifference 508 is less than or equal to a threshold performance score.In such a scenario, model performance subsystem 116 may generate andoutput an indicator 512 indicating that a performance of the currentversion of the machine learning model has not changed by more than athreshold amount with respect to a previous version’s performance. Insome cases, indicator 512 being output may indicate that a currentmachine learning model does not need to be retrained, rebuilt, orreplaced. As another example, one of conditions 510 may be satisfied ifperformance difference 508 is greater than a threshold performancescore. In such a scenario, model performance subsystem 116 may generateand output an indicator 514 indicating that a performance of the currentversion of the machine learning model has changed by more than thethreshold amount with respect to the previous version’s performance. Insome cases, indicator 514 being output may indicate that a currentversion of the machine learning model should be retrained, rebuilt, orreplaced.

In some embodiments, one of conditions 510 may be satisfied ifperformance score 504 is less than or equal to a threshold performancescore. For example, if performance score 504 is less than a thresholdperformance score, then model performance subsystem 116 may generate andoutput indicator 514 (e.g., the machine learning model is to beretrained, rebuilt, or replaced). As another example, that one ofconditions 510 not being satisfied, or a different condition beingsatisfied, includes performance score 504 being greater than thethreshold performance score, then model performance subsystem 116 maygenerate and output indicator 512 (e.g., the machine learning model maynot need to be retrained, rebuilt, or replaced).

Production data, such as production data 202, may include a plurality ofdata items representing one or more feature sets. For example,production data 202 may include a stream of credit card applications,and each credit card application may include information such as anapplicant’s annual salary, residence, employment history, and the like.The information included by each credit card application may represent afeature set, and thus the production data may include a plurality ofdata items representing various feature sets. In some embodiments, theproduction data may include feature sets, however it may not includeobserved results corresponding to the feature sets. In some embodiments,the production data may include feature sets and observed resultscorresponding to the feature sets, however the observed results may bemasked so as to not be input to the machine learning model (e.g., one ofmachine learning models 410 a-410 n).

As mentioned above, machine learning models 410 a-410 n may beconfigured to take, as input, production data 202 and generate outputdatasets 412 a-412 n, respectively, which may be generated based on thefeature sets included in production data 202. Output datasets 412 a-412n may represent predictions from a respective one of machine learningmodels 410 a-410 n for each feature set input to that machine learningmodel. In some embodiments, model performance subsystem 116 may beconfigured to compute performance score 504 for a corresponding machinelearning model (e.g., one or more of machine learning models 410 a-410n) based on the output datasets (e.g., a respective one or more ofoutput datasets 412 a-412 n) and production data 202. For example, modelperformance subsystem 116 may determine performance difference 508between observed results and output datasets 412. Based on thedifference and a number of feature sets included by production data 202,performance score 504 of a corresponding machine learning model may bedetermined.

In some embodiments, model performance subsystem 116 may determinewhether the computed accuracy score for the machine learning modelsatisfies one or more of conditions 510. For example, one of conditions510 may be a threshold accuracy condition satisfied if performance score504 is less than a threshold performance score, as mentioned above. Asanother example, the threshold accuracy condition may be satisfied ifperformance score 504 is greater than or equal to a threshold accuracyscore. In some embodiments, the threshold performance score may bedetermined based on prior performance score 506 previously determinedfor a corresponding machine learning model during the training process.For instance, during training, a machine learning model may have beendetermined to have an accuracy score of S1 based on validation data. Thethreshold performance score for the machine learning model based onproduction data 202 may be determined using accuracy score S1. In someembodiments, the threshold performance score may be the same or similarto accuracy score S1. For example, if accuracy score S1 is 80% (e.g.,indicating that, during training, the machine learning modelsuccessfully predicted 80% of the results of the test data), thethreshold performance score may be 80% +/- δ, where δ is a configurablevalue depending on the particular machine learning model. For example, δmay be 2% or more, 5% or more, 10% or more, or other values.

Some embodiments include using some of production data 202 to generateupdated training data. In some embodiments, the updated training data tore-train the machine learning model that produced output datasets 412,to generate a new instance of that machine learning model, or togenerate a new machine learning model.

In some embodiments, model performance subsystem 116 may be configuredto compute residuals between predicted results and observed results,such as observed results from production data 202. Residuals represent adifference between what is actually detected and what is predicted. Forexample if, for a machine learning model configured to predict a creditscore for a given credit application, a predicted credit score is 700and an actual credit score is 750, then the residual would be 50. Insome embodiments, a graphical representation of the residuals may begenerated to identify which feature or features contribute most or leastto residuals. For example, the residuals may indicate that geographicallocation affects a credit score greater than expected or desired. Insuch cases, the machine learning model may, during a rebuild orsubsequent training, or during deployment, modify one or more parameters(e.g., hyperparameters) to decrease or increase the effect ofgeographical location on credit score predictions. In some embodiments,an accuracy of the trained machine learning model may be determinedbased on the residuals. For instance, because the residuals reflect thedifference between the predicted results and the observed results for amachine learning model, the accuracy score of the machine learning modelmay also be determined based on the residuals.

In some embodiments, a contribution to the residuals for each feature ofa plurality of features represented by the production data may bedetermined. For example, a feature set, representing features F1 and F3,may be associated with an observed result. Furthermore, the feature setmay, when input to a machine learning model, cause the machine learningmodel to produce predicted result (e.g., included in output datasets412). A contribution to the residual (e.g., the difference between thepredicted result and the observed result) for both features F1 and F3may be determined to identify whether feature F1 and/or feature F3contribute to the residuals by more than a threshold amount. In someembodiments, if it is determined that a contribution to the residuals ofone or more of the features included by the feature sets of theproduction data is equal to or greater than a threshold amount ofcontribution for residuals, model performance subsystem 116 may generateindicator 514 to cause the training data to be updated and/or cause themachine learning model to be rebuilt. As an example, the thresholdamount of contribution for the residuals may be greater than 5%, greaterthan 10%, greater than 20%, or other amounts. For instance, if featureF1′s contribution to the residuals is greater than the threshold amountof contribution (e.g., feature F1 contributes to the residuals by morethan 5%), then this may indicate that training data should be updated,and/or the machine learning model should be rebuilt.

Depending on the configuration of model performance subsystem 116, oneor more of conditions 510 being satisfied may be sufficient to causeindicator 512 or 514 to be generated. Thus, although multiple scenariosare described above, persons of ordinary skill in the art will recognizethat each of conditions 510 need not be satisfied in order for computersystem 102 to perform a subsequent action.

In some embodiments, replacing the machine learning model may includecausing a new machine learning model to be built. The new machinelearning model may have similar features as the original machinelearning model (e.g., same or similar hyperparameters, executionfrequency, model type, etc.), or the features may differ. For example,performance score 504 may be generated based on datasets 412, generatedby one of machine learning models 410 a-410 n. If each of machinelearning models 410 a-410 n have a same execution frequency (e.g.,weekly execution frequency), then in response to determining that aparticular one or more of conditions 510 has/have been satisfied, modelperformance subsystem 116 may determine that a new machine learningmodel having a different execution frequency (e.g., monthly executionfrequency) is to be built. In some embodiments, replacing the machinelearning model may include promoting one machine learning model frombeing a secondary machine learning model to be a primary machinelearning model, while demoting the previous primary machine learningmodel to now be a secondary machine learning model. For example, if twoor more machine learning models are part of a champion-challengerscenario, then satisfying one of conditions 510 may include making oneof the challenger machine learning models the champion and making theprevious champion machine learning model a challenger.

Some embodiments include performing a side-by-side comparison of onemachine learning model against another machine learning model (ormultiple machine learning models). For example, a performance score ofmachine learning model 410 a may be compared in parallel to aperformance score of machine learning model 410 b. As different machinelearning models take, as input, different parameters, one machinelearning model performing poorly on the production data may indicatethat that machine learning model needs to be rebuilt, retrained, orreplaced because of the types of features included in the productiondata. For example, machine learning model 410 a may employ a first typeof feature to generate datasets 412 a while machine learning model 410 bmay employ a second type of feature to generate datasets 412 b. If aperformance score for machine learning model 410 a is greater than aperformance score for machine learning model 410 b, such as by more thana threshold amount, then this may indicate that the production dataincludes irregularities, Null sets, not enough instances of, or otherissues, with respect to the second type of feature. If it is determinedthat a particular type of feature included in the production data causesproblems, and that some machine learning models take, as input, thattype of feature, then those machine learning models may be avoided orreplaced with other machine learning models that do not take, as input,that type of feature.

In some embodiments, model build subsystem 118 may be configured tobuild, train, or facilitate replacement of one or more machine learningmodels. For example, if model performance subsystem 116 outputsindicator 514, indicating that a new machine learning model is to bebuilt based on another machine learning model satisfying one ofconditions 510, model build subsystem 118 may facilitate building thenew machine learning model. As an example, with reference to FIG. 6 ,process 600 describes an example for building a machine learning modelin response to determining that a performance score of a differentmachine learning model satisfies one or more conditions. As seen FIG. 6, process 600 includes model build subsystem 118 receiving indicator 514generated and output by model performance subsystem 116. Upon receipt ofindicator 514, model build subsystem 118 may determine a type of machinelearning model to be built and data to be retrieved to train andvalidate the machine learning model to be built. Some embodimentsinclude obtaining a design specification indicating a type of machinelearning model to be built, a source with which data for training themachine learning model is to be retrieved from, model parameters to betuned by the training process, a number of machine learning models to bebuilt, and/or other information.

In some embodiments, the model parameters represent features or types offeatures for the machine learning model to be trained. A featurerepresents a variable that serves as an input to a model and is used bythe model to make predictions. In some embodiments, features may beorthogonal to one another. For example, each feature may occupy adimension of an n-dimensional feature space. The model parameters, insome cases, may indicate the types of features represented by data usedto train the machine learning model, as well as the type of featuresexpected to be represented by the production data input to the trainedmachine learning model. As an example, data including features, such asnoise ratios, lengths of sound, relative power, etc., may serve as aninput to a prediction model related to recognizing phonemes for speechrecognition processes. As another example, data including features suchas edges, objects, pixel information, may serve as an input to aprediction model related to computer vision analysis. As still yetanother example, data including features, such as income, credit score,and biographical information may serve as an input to a prediction modelrelated to financial applications. Each of the features (e.g., noiserations, lengths of sound, relative power, edges, objects, income,credit score, biographical information, or other features) may bedifferent types of features. The feature type may relate to the genre ofthe prediction model (e.g., speech recognition models, computer visionmodels, etc.) or the different individual fields encompassed by afeature (e.g., length of sounds in units of time, income in units ofdollars, etc.). As described herein, a feature type corresponds to atype of feature (e.g., what the feature represents). For example, thefeature type “salary information” may correspond to the feature“salary,” which may be used as a model input parameter to afinancially-related prediction model. In some embodiments, the modelparameters may also indicate hyperparameters associated with the trainedmachine learning model. A hyperparameter represents a configurablevariable whose value is estimated by a model based on input data. As anexample, for a PCA model, a number of components to keep represents onetype of hyperparameter.

The model parameters may indicate distributions, trends, value ranges,or other aspects, of the features included within data to be input to amachine learning model. In some embodiments, training data used to trainthe machine learning model may have a particular distribution offeatures (e.g., the training data includes a first percentage of a firsttype of feature, a second percentage of a second type of feature, and soon). Based on the particular distribution of features of the trainingdata, production data to be input to a trained machine learning model isexpected to also include a same or similar distribution of features. Ifso, then the trained machine learning model should perform accuratepredictions. However, if the distribution of features of data to beinput to the trained machine learning model differs from thedistribution of features of included in the training data used to trainthe machine learning model, then the predictions of the trained machinelearning model may not be accurate.

The type of model to be built may be specified based on one or more of atype of machine learning model used to produce datasets 412 with whichcaused indicator 514 to be produced, a type of data to be analyzed upondeployment of the model, or other factors. For instance, indicator 514may include metadata specifying the type of model that produced datasets412. Based on the design specification, model build subsystem 118 mayretrieve computer code, software, scripts, or other data needed to buildthe new machine learning model from model database 134. Furthermore,based on the design specification, model build subsystem 118 mayretrieve build data from training data database 136 to be used to buildthe new machine learning model. The build data may include training dataand validation data. In some embodiments, the build data may begenerated based on historical data, such as previous production datathat machine learning models have been executed on. The training data isused to train the machine learning model, such as tuning modelparameters to specific values. The validation data is used to determinehow well the model has been trained. For example, the validation datacan include known results and inputs that, if the model is trainedaccurately, are expected to produce the known results. In someembodiments, training may be completed when the accuracy of the model,tested with the validation data, equals or exceeds an accuracythreshold. For instance, model build subsystem 118 may output machinelearning model 610 for deployment or further analysis in response todetermining that the training process has been completed. Machinelearning model 610 may also be stored in model database 134 forretrieval and use during subsequent deployment cycles.

Models stored in model database 134 be trained, or re-trained,periodically, such as every day, every week, every month, or at otherintervals. Model build subsystem 118 may retrieve training data fromtraining data database 136 for training a given machine learning model,where the particular training data used may vary depending on themachine learning model that is being trained. For example, an objectrecognition model may be trained using images of objects. As anotherexample, a credit scoring model may be trained using creditapplications. Some embodiments include training data having labels foreach data item indicating an expected result for the machine learningmodel. For example, an object recognition model may be trained usingimages of objects including labels indicating a category of object thateach image represents. This may allow the machine learning model toadaptively “learn” by computing residuals between the predicted resultsand the observed results (e.g., the results that would have beenobtained if the machine learning model operated without error), andsubsequently altering values of various hyperparameters of the machinelearning model to try and minimize the residuals.

The model build process performed by model build subsystem 118 includesmultiple steps including (1) a data pull process; (2) a featureengineering process; (3) a model build process; and (4) a model scoringprocess. In some embodiments, an additional data splitting process maybe performed, as described below.

The data pull process may include retrieving the training data fromtraining data database 136 for training a machine learning model. Insome embodiments, build data may be generated from the training dataretrieved from training data database 136. The training data may besplit to segment the retrieved data into training data and validationdata, which is also referred to herein interchangeably as test data. Thetraining data may be used to train the machine learning model, whereasthe validation data may be used to test an accuracy of the trainedmachine learning model. The data splitting process generally includesselecting at least some of data sets from training data database 136 anddesignating some of the selected data sets as training data anddesignating other of the selected data sets as validation data. In someembodiments, the training data and the validation data may be labeleddata (e.g., data items including labels representing expected outcomes).For example, a data item included by the training data may represent acredit card application and the label associated with the data item maybe an indication of whether the credit card application should beapproved or denied. In some embodiments, the labels may be metadataassociated with the training data and the validation data.

In some embodiments, the feature engineering process may includerefining the training data such that the data represents features neededfor input to the machine learning model to be trained. The featureengineering process may also refine the validation data in a similarmanner. In some embodiments, the feature engineering process may usedomain knowledge associated with the machine learning model to betrained to extract features from the training data relevant to themachine learning model. The extracted features can be used to improvethe performance of the machine learning algorithms.

In some embodiments, the model build process may include training aselected machine learning model with the training data. The model buildprocess may take the training data as inputs for the selected machinelearning model, which may provide outputs that can be fed back to themachine learning model as input to train the machine learning model(e.g., alone or in conjunction with user indications of the accuracy ofthe outputs, labels associated with the inputs, or with other referencefeedback information). In some embodiments, the model build process maycause, or otherwise facilitate, the machine learning model to update itsconfigurations (e.g., weights, biases, or other parameters) based on itsassessment of its prediction and reference feedback information (e.g.,user indication of accuracy, reference labels, or other information). Insome embodiments, where the machine learning model is a neural network,connection weights may be adjusted to reconcile differences between theneural network’s prediction and the reference feedback. Some embodimentsinclude one or more neurons (or nodes) of the neural network requiringthat their respective errors be sent backward through the neural networkto them to facilitate the update process (e.g., backpropagation oferror). Updates to the connection weights may, for example, bereflective of the magnitude of error propagated backward after a forwardpass has been completed. In this way, for example, the machine learningmodel may be trained to generate better predictions.

In some embodiments, the model scoring process may include testing theaccuracy of the built machine learning model to determine whether themachine learning model has been properly trained. For example, the modelscoring process may cause the built machine learning model to take, asinput, the validation data, and may compare the outputs of the builtmachine learning model to the results indicated by the labels associatedwith validation data. If the model scoring process does not yieldpositive results the machine learning model may be re-trained with newtraining data and scored, with this process repeating until the model iscapable of accurately predicting results for the validation data (or newvalidation data). For example, the model scoring process may compute anaccuracy score for the predicted outputs of the built machine learningmodel based on a comparison of the outputs from the built machinelearning model and the results stored as labels with the validationdata. If the accuracy score of the built machine learning modelsatisfies a threshold training condition, such as the accuracy scorebeing greater than or equal to a threshold training score, then themodel scoring process may output the trained machine learning model.

FIG. 7 shows a database storing machine learning models having variousexecution frequencies, in accordance with one or more embodiments. Modeldatabase 134 may include multiple sets of machine learning models, eachof which may have a different execution frequency, purpose. For example,model database 134 includes a first set of machine learning models 702a-702 n, each having a first execution frequency, F1, and may alsoinclude a second set of machine learning models 704 a-704 m, each havinga second execution frequency, F2. For example, first execution frequencyF1 may be an hourly, daily, weekly, or other frequencies with whichmachine learning models 702 a-702 n execute. Second execution frequencyF2 may be weekly, monthly, bimonthly, quarterly, yearly, or otherfrequencies with which machine learning models 704 a-704 m execute.Although only two sets of machine learning models having two differentexecution frequencies are included in model database 134, additionalsets of machine learning models having different execution frequenciesmay be stored in model database 134. The number of machine learningmodels included in the first set of machine learning models may be thesame or different than the number of machine learning models included inthe second set of machine learning models. For example, each set ofmachine learning models may include one or more machine learning models,two or more machine learning models, ten or more machine learningmodels, or other numbers of machine learning models.

In some embodiments, when a changepoint has been detected in productiondata, a particular set of machine learning models may be executed on theproduction data. For example, machine learning models 702 a-702 n may beexecuted on the production data in response to a changepoint beingdetected. In some embodiments, in response to determining that aperformance score of one or more of machine learning models 702 a-702 nis less than a threshold performance score, one or more of machinelearning models 704 a-704 m may be executed on the production data.Newly built or newly trained models may also be stored in model database134. In some embodiments, the newly built models may be stored withother machine learning models having a same or similar executionfrequency. For example, if a newly built machine learning model, such asone built in response to the performance score of one or more of machinelearning models 702 a-702 n being less than a threshold performancescore, has an execution frequency F2, then the newly built machinelearning model may be added to machine learning models 704 a-704 m.

Example Flowcharts

FIGS. 8A-8B and 9 are example flowcharts of processing operations ofmethods that enable the various features and functionality of the systemas described in detail above. The processing operations of each methodpresented below are intended to be illustrative and non-limiting. Insome embodiments, for example, the methods may be accomplished with oneor more additional operations not described, and/or without one or moreof the operations discussed. Additionally, the order in which theprocessing operations of the methods are illustrated (and describedbelow) is not intended to be limiting.

In some embodiments, the methods may be implemented in one or moreprocessing devices (e.g., a digital processor, an analog processor, adigital circuit designed to process information, an analog circuitdesigned to process information, a state machine, and/or othermechanisms for electronically processing information). The processingdevices may include one or more devices executing some or all of theoperations of the methods in response to instructions storedelectronically on an electronic storage medium. The processing devicesmay include one or more devices configured through hardware, firmware,and/or software to be specifically designed for execution of one or moreof the operations of the methods.

FIGS. 8A and 8B show flowcharts of a method 800 for determining amachine learning model to execute based on results of other machinelearning models, in accordance with one or more embodiments. Method 800may begin at an operation 802. In operation 802, production data to beprovided to at least a first machine learning model and a second machinelearning mode, each having a first execution frequency, may be obtained.In some embodiments, the first machine learning model and the secondmachine learning model are selected based on each having a firstexecution frequency (e.g., executing daily, weekly, etc.). In someembodiments, the first machine learning model and the second machinelearning model are part of, or form, a set of machine learning modelsthat are to initially be executed on production data. The productiondata obtained may also be provided to other machine learning modelshaving a different execution frequency (e.g., monthly, quarterly, etc.).In some embodiments, the production data may be obtained via a datafeed, such as data feed 140. The data feed may be configured to receiveupdated application data from one or more real-time applications. Insome embodiments, the production data may be stored in production datadatabase 132 prior to, in parallel to, or after being provided tocomputer system 102. The production data may be retrieved in some casesfrom production data database 132 instead of from data feed 140. In someembodiments, operation 802 may be performed by a subsystem that is thesame or similar to changepoint detection subsystem 112.

In an operation 804, the production data may be analyzed forchangepoints. A changepoint represents instances of data abruptlyshifting in some manner. In particular, changepoints may representabrupt shifts in time series data (e.g., data that is sequential intime). A goal of changepoint detection is to identify a location (e.g.,a time) that a particular changepoint or changepoints occurred in thedata, as well as determining a number of changepoints in the data. Theproduction data, such as production data 202, may represent time seriesdata as described above with respect to Equation 1. The time series datamay be multi-dimensional, including multiple variables (e.g., a price ofan item, a salary, a credit score, a grayscale value, etc.), some ofwhich may be independent variables, while others may be correlated.Thus, changepoints, which represent abrupt changes in a value of thedata, may occur in one or more dimensions. In some embodiments,analyzing the production data for changepoints may include identifyingcandidate multiple changepoint configurations in the production data202, which may each be referred to as a model or candidate model. Modelselection may be based on the BDML framework, and may include computinga BDML score for each candidate model and selecting the candidate modelhaving a smallest BDML score. All changepoints included in the selectedcandidate model may be classified as detected changepoints. Someexamples of changes that can occur are shifts in a mean value of thedata and/or shifts in a slope of the data. For instance, the productiondata can include different regimes which will have one mean and slopevalue before a given changepoint and a different mean and slope valueafter the changepoint. In some embodiments, operation 804 may beperformed by a subsystem that is the same or similar to changepointdetection subsystem 112.

In an operation 806, a determination may be made as to whether anychangepoints have been detected within the production data. In someembodiments, changepoints may be detected using the BDML framework,however other frameworks may be used in addition to or in lieu of theBDML framework. For example, an object BDML framework or “oBDML”framework may be used, an MDL framework, or a BIC framework may be used.Additional details regarding the various techniques capable of beingused to detect changepoints within multivariant data, such as productiondata as detailed herein, is described in Li et al., “MultipleChangepoint Detection with Partial Information on Changepoint Times,”2019, the disclosure of which is incorporated herein by reference in itsentirety. In some embodiments, operation 806 may be performed by asubsystem that is the same or similar to changepoint detection subsystem112.

If, at operation 806, it is determined that no changepoints have beendetected in the production data, then method 800 may proceed tooperation 808. In operation 808, the first machine learning model may becaused to execute on the production data in lieu of the second machinelearning model. For example, as mentioned above, the production dataobtained at operation 802 is for the first machine learning model andthe second machine learning model, which both have a same executionfrequency (e.g., a first execution frequency). Prior to the productiondata being provided to, or executed on by, the first or second machinelearning model, the changepoint detection processes may be performed todetermine whether the production data includes any instances of achangepoint. After determining that no changepoints are detected in theproduction data, the production data may then be provided to, andexecuted on by, the first machine learning model. In some embodiments,if no changepoints are detected in the production data, the productiondata may still be provided to the second machine learning model, howeverthe second machine learning model may not execute on the production data(e.g., may be prevented from executing). In some embodiments, operation808 may be performed by a subsystem that is the same or similar tochangepoint detection subsystem 112.

If, however, at operation 806, it is determined that one or morechangepoints are detected in the production data, method 800 may proceedto operation 810. In operation 810, the first machine learning model andthe second machine learning model may be caused to execute on theproduction data. The first machine learning model executing on theproduction data may cause first datasets to be obtained, and the secondmachine learning model executing on the production data may cause thesecond datasets to be obtained. For example, in response to detecting atleast one changepoint in production data 202, machine learning model 410a may be executed on production data 202 to obtain output datasets 412 aand machine learning model 410 b may be executed on production data 202to obtain output datasets 412 b. In some embodiments, operation 810 maybe performed by a subsystem that is the same or similar to modelexecution subsystem 114.

In some embodiments, method 800 may proceed from operation 810 tooperation 812, depicted in FIG. 8B.

In operation 812, a first performance score may be computed for thefirst machine learning model and a second performance score may becomputed for the second machine learning model. In some embodiments, theperformance scores may be computed based on the output datasetsgenerated by a corresponding machine learning model. For example, afirst performance score (e.g., performance score 504) may be computedfor machine learning model 410 a based on output datasets 412 a, and asecond performance score may be computed for machine learning model 410b based on output datasets 412 b. In some embodiments, a type ofperformance metric may be selected in advance of the performance scorebeing computed, where the type of performance metric is selected basedon a type of machine learning model that the corresponding outputdatasets were produced by, a goal of the machine learning model, orother factors. The performance metrics that may be computed to obtainthe performance score may include a mean, variance, bias, or otherperformance metric. In some embodiments, operation 812 may be performedby a subsystem that is the same or similar to model performancesubsystem 116.

In an operation 814, a determination may be made as to whether the firstor second performance is less than a threshold performance score. Insome embodiments, the threshold performance score may be determinedbased on previous performance scores for the first machine learningmodel, the second machine learning model, other machine learning models,combinations of machine learning models, or other performance scores.The threshold performance score may be configured by a systemadministrator (e.g., a user of client device 104, computer system 102,or other components of system 100), and may be dynamically adjustedduring operation. If the performance score of either the first machinelearning model or the second machine learning model is less than thethreshold performance score, this may indicate that the production datamay cause problems to arise if executed on by other machine learningmodels. For example, other machine learning models have differentexecution frequencies than those of the first and second machinelearning models. Thus, detecting performance issues with the first andsecond machine learning models for the production data can help preventproblems arising with other machine learning models that would otherwiseexecute on the production data, even if those machine learning modelshave not done so yet. In some embodiments, operation 814 may beperformed by a subsystem that is the same or similar to modelperformance subsystem 116.

If, at operation 814, it is determined that either the first or secondperformance score is less than the threshold performance score, thenmethod 800 may proceed to operation 816. In operation 816, a thirdmachine learning model may be built or caused to be built. The thirdmachine learning model is to have a second execution frequency differentthan the first execution frequency, and is to be executed on theproduction data. For example, the third machine learning model may bebuilt to have an execution frequency of monthly (e.g., executes monthly)whereas the first and second machine learning models may have anexecution frequency of daily (e.g., execute daily). In some embodiments,operation 816 may be performed by a subsystem that is the same orsimilar to model build subsystem 118.

In operation 818, a fourth machine learning model, having the secondexecution frequency, may be prevented from being executed on theproduction data. By building the third machine learning model to executeon the production data while also preventing the fourth machine learningfrom executing on the production data, issues that would otherwise arisewith the fourth machine learning model are mitigated, thereby savingprecious computational resources that can be reallocated to other tasks.In some embodiments, operation 818 may be performed by a subsystem thatis the same or similar to model execution subsystem 114, model buildsubsystem 118, or a combination of model execution subsystem 114 andmodel build subsystem 118.

If, at operation 814, it is determined that neither the first nor secondperformance score is less than the threshold performance score, thenmethod 800 may proceed to operation 820. At operation 820, theproduction data may be provided to the fourth machine learning model.The fourth machine learning model, as mentioned above, may have a secondexecution frequency different from the execution frequency of the firstand second machine learning models. Therefore, the fourth machinelearning model may not execute on the production data at a same time asthat of the first and second machine learning models. Furthermore, ifthe first and second performance scores are not less than the thresholdperformance, then this indicates that the production data will not causeproblems when executed on by the fourth machine learning model, andtherefore an additional machine learning model may not be necessary. Insome embodiments, operation 820 may be performed by a subsystem that isthe same or similar to model execution subsystem 114, model buildsubsystem 118, or a combination of model execution subsystem 114 andmodel build subsystem 118.

FIG. 9 shows a flowchart of a method 900 for assigning machine learningmodels to be a primary model or a secondary model, in accordance withone or more embodiments. Method 900 may begin at an operation 902. Priorto operation 902, some of the same or similar operations included inmethod 800 may be performed. For instance, operations 802-816 may beperformed and, subsequent to operation 816, operation 902 may beperformed. As an example, in response to building a new (third) machinelearning model having a second execution frequency (different than thefirst execution frequency of the first and second machine learningmodels of method 800), which is to be executed on the production data,operation 902 of method 900 may be performed.

In operation 902, the third machine learning model (e.g., the newlybuilt machine learning model of operation 816) may be assigned as aprimary model and the fourth machine learning model may be assigned tobe a secondary model. In some embodiments, the primary model, which mayalso be referred to as a “champion” model, may be designated forgenerating prediction data for a particular use case. For example, thechampion model may be used to generate prediction data for a businessuse, decision making process, or other purpose. The secondary model,which may also be referred to as a “challenger” model, may be designatedfor generating prediction data for evaluation. The prediction data forevaluation may not necessarily be used for a same purpose as that of theprediction data generated by the primary, or champion, model. In somecases, multiple secondary models may be included in the productionenvironment, each to be executed on with the production data, and eachof which may be configured to generate prediction data for variousevaluation purposes. In some embodiments, prior to the third machinelearning model being assigned to be the primary model, a differentmachine learning model (e.g., the fourth machine learning model) may beassigned to be the primary model. Some embodiments include selectingwhich model is to be the primary model, and also which model or modelsare to be the secondary models, based on prior performances during priordeployments, accuracy scores computed during training of the models, theuse case with which the models are to generate prediction data for, orfor other reasons. Additionally, the primary and secondary models mayhave a same or different execution frequency. In some embodiments,operation 902 may be performed by a subsystem that is the same orsimilar to model execution subsystem 114.

In an operation 904, the third machine learning model and the fourthmachine learning model may be caused to execute on the production data.For example, the third machine learning model (e.g., the primary model)and the fourth machine learning model (e.g., the secondary model) mayexecute on production data 202. In some embodiments, third outputdatasets and fourth output datasets may be obtained based on the thirdmachine learning model and the fourth machine learning model,respectively, executing on the production data. In some embodiments,operation 904 may be performed by a subsystem that is the same orsimilar to model execution subsystem 114.

In an operation 906, a third performance score and a fourth performancescore may be computed for the third machine learning model and thefourth machine learning model, respectively. Similar to operation 812, atype of performance metric may be selected in advance of the performancescore being computed, where the type of performance metric is selectedbased on a type of machine learning model that the corresponding outputdatasets were produced by, a goal of the machine learning model, orother factors. The performance metrics that may be computed may includea mean, variance, bias, or other performance metric. In someembodiments, operation 906 may be performed by a subsystem that is thesame or similar to model performance subsystem 116.

In an operation 908, a determination may be made as to whether the thirdperformance score is greater than the fourth performance score. Forexample, if the third performance score of the third machine learningmodel is represented as S3 and the fourth performance score of thefourth machine learning model is represented as S4, then a determinationis made as to whether S3 > S4. In some embodiments, if S3 = S4, thenthis may be considered the same scenario as if S3 is greater than S4.Alternatively, if S3 = S4, then this may be considered the same scenarioas if S4 is greater than S3. Although only two performance scores, S3and S4, of two machine learning models are considered in the example ofmethod 900, persons of ordinary skill in the art will recognize that ifadditional machine learning models are included, then similarcomparisons may be performed at operation 908. For example, if a fifthmachine learning model is included, which may also be a “secondary”model, then a determination may be made as to whether the thirdperformance score is greater than the fourth performance score and thefifth performance score (e.g., a performance score computed based on thefifth machine learning model executing on the production data). However,for simplicity, operation 908 considers two machine learning models. Insome embodiments, operation 908 may be performed by a subsystem that isthe same or similar to model performance subsystem 116.

If, at operation 908, it is determined that the third performance scoreis greater than the fourth performance score, then method 900 mayproceed to operation 910. In operation 910, the third machine learningmodel may be kept as the primary model and the fourth machine learningmodel may be kept as the secondary model. This may allow for the thirdmachine learning model, which is determined to perform more accuratelywith respect to the production data, as a source for generatingprediction data for the particular use of system 100. In someembodiments, operation 910 may be performed by a subsystem that is thesame or similar to model execution subsystem 114, model performancesubsystem 116, or a combination of model execution subsystem 114 andmodel performance subsystem 116.

However, if at operation 910 it is determined that the third performancescore is not greater than the fourth performance score, then method 900may proceed to operation 912. At operation 912, the fourth machinelearning model may be assigned as the primary model and the thirdmachine learning model may be assigned as the secondary model. Thus, thefourth machine learning model may be used to generate prediction datafor analyzing particular use cases, such as the effectiveness of aparticular strategy, service, or other scenario, and the third machinelearning model may be used to generate prediction data for evaluationpurposes. In some embodiments, operation 910 may be performed by asubsystem that is the same or similar to model execution subsystem 114,model performance subsystem 116, or a combination of model executionsubsystem 114 and model performance subsystem 116.

Various aspects of the disclosed embodiments may be implemented viasoftware modules executed directly or, alternatively, using machinelearning as a service platform. For example, some embodiments includemaking an API call where production data is passed to the service andthe outputs of one or more processes are returned.

In some embodiments, the various computers and subsystems illustrated inFIG. 1 may include one or more computing devices that are programmed toperform the functions described herein. The computing devices mayinclude one or more electronic storages (e.g., database(s) 130, whichmay include production data database 132, model database 134, trainingdata database 136, performance database 138, etc., or other electronicstorages), one or more physical processors programmed with one or morecomputer program instructions, and/or other components. It should benoted that although the illustrated embodiments include a singleinstance of production data database 132, model database 134, trainingdata database 136, performance database 138, multiple instances of eachdatabase may be employed. The computing devices may includecommunication lines or ports to enable the exchange of information withone or more networks (e.g., network(s) 150) or other computing platformsvia wired or wireless techniques (e.g., Ethernet, fiber optics, coaxialcable, WiFi, Bluetooth, near field communication, or othertechnologies). The computing devices may include a plurality ofhardware, software, and/or firmware components operating together. Forexample, the computing devices may be implemented by a cloud ofcomputing platforms operating together as the computing devices.

The electronic storages may include non-transitory storage media thatelectronically stores information. The storage media of the electronicstorages may include one or both of (i) system storage that is providedintegrally (e.g., substantially non-removable) with servers or clientdevices or (ii) removable storage that is removably connectable to theservers or client devices via, for example, a port (e.g., a USB port, afirewire port, etc.) or a drive (e.g., a disk drive, etc.). Theelectronic storages may include one or more of optically readablestorage media (e.g., optical disks, etc.), magnetically readable storagemedia (e.g., magnetic tape, magnetic hard drive, floppy drive, etc.),electrical charge-based storage media (e.g., EEPROM, RAM, etc.),solid-state storage media (e.g., flash drive, etc.), and/or otherelectronically readable storage media. The electronic storages mayinclude one or more virtual storage resources (e.g., cloud storage, avirtual private network, and/or other virtual storage resources). Theelectronic storage may store software algorithms, information determinedby the processors, information obtained from servers, informationobtained from client devices, or other information that enables thefunctionality as described herein.

The processors may be programmed to provide information processingcapabilities in the computing devices. As such, the processors mayinclude one or more of a digital processor, an analog processor, adigital circuit designed to process information, an analog circuitdesigned to process information, a state machine, and/or othermechanisms for electronically processing information. In someembodiments, the processors may include a plurality of processing units.These processing units may be physically located within the same device,or the processors may represent processing functionality of a pluralityof devices operating in coordination. The processors may be programmedto execute computer program instructions to perform functions describedherein of subsystems 112-118 or other subsystems. The processors may beprogrammed to execute computer program instructions by software;hardware; firmware; some combination of software, hardware, or firmware;and/or other mechanisms for configuring processing capabilities on theprocessors.

It should be appreciated that the description of the functionalityprovided by the different subsystems 112-118 described herein is forillustrative purposes, and is not intended to be limiting, as any ofsubsystems 112-118 may provide more or less functionality than isdescribed. For example, one or more of subsystems 112-118 may beeliminated, and some or all of its functionality may be provided byother ones of subsystems 112-118. As another example, additionalsubsystems may be programmed to perform some or all of the functionalityattributed herein to one of subsystems 112-118.

Although example embodiments have been described in detail for thepurpose of illustration, it is to be understood that such detail issolely for that purpose and that embodiments are not limited to thedisclosed embodiments, but, on the contrary, are intended to covermodifications and equivalent arrangements that are within the scope ofthe appended claims. For example, it is to be understood thatembodiments contemplate that, to the extent possible, one or morefeatures of any embodiment can be combined with one or more features ofany other embodiment.

As used throughout this application, the word “may” is used in apermissive sense (i.e., meaning having the potential to), rather thanthe mandatory sense (i.e., meaning must). The words “comprise,”“comprising,” “comprises,” “include”, “including”, and “includes” andthe like mean including, but not limited to. As used throughout thisapplication, the singular forms “a,” “an,” and “the” include pluralreferents unless the context clearly indicates otherwise, andnotwithstanding the use of other terms and phrases for one or moreelements, such as “one or more.” The term “or” is non-exclusive (i.e.,encompassing both “and” and “or”), unless the context clearly indicatesotherwise. Further, unless otherwise indicated, statements that onevalue or action is “based on” another condition or value encompass bothinstances in which the condition or value is the sole factor andinstances in which the condition or value is one factor among aplurality of factors. Unless the context clearly indicates otherwise,statements that “each” instance of some collection have some propertyshould not be read to exclude cases where some otherwise identical orsimilar members of a larger collection do not have the property, i.e.,each does not necessarily mean each and every.

Additional example embodiments are provided with reference to thefollowing enumerated embodiments:

1. A method, comprising: obtaining first data from a data feed to beprovided to a plurality of machine learning models; detecting achangepoint in the first data; responsive to the changepoint beingdetected, causing a first machine learning model to be executed on thefirst data to obtain first output datasets; computing a firstperformance score for the first machine learning model based on thefirst output datasets; and causing a second machine learning model toexecute on the first data based on the first performance scoresatisfying a first condition.

2. The method of embodiment 1, further comprising: in response todetermining that the first performance score satisfies the firstcondition, building the second machine learning model.

3. The method of embodiment 2, wherein building the second machinelearning model comprises: obtaining build data to be used to build thesecond machine learning model, the build data comprising training dataand validation data; selecting a type of machine learning with which thesecond machine learning model is to be; training a machine learningmodel using the training data to obtain a trained machine learningmodel; and determining, based on the validation data, whether thetrained machine learning model has an accuracy score greater than orequal to a threshold accuracy score, wherein the trained machinelearning model is capable of being used as the second machine learningmodel in response to determining that the accuracy score is greater thanor equal to the threshold accuracy score

4. The method of any one of embodiments 1-3, wherein: the first machinelearning model has a first execution frequency; the second machinelearning model has a second execution frequency; and the secondexecution frequency is less than the first execution frequency.

5. The method of embodiment 4, wherein: the first execution frequencycomprises an execution frequency of hourly, daily, weekly, or monthly;and the second execution frequency comprises an execution frequencycomprises an execution frequency of weekly, monthly, quarterly, orannually such that the second execution is less frequent than the firstexecution frequency.

6. The method of embodiment 4, wherein the first execution frequency isa weekly execution frequency and the second execution frequency is amonthly execution frequency.

7. The method of any one of embodiments 1-6, wherein obtaining the firstdata comprises: obtaining the first data via a data feed configured toreceive updated application data from one or more real-timeapplications.

8. The method of embodiment 7, wherein the data feed is selected from aplurality of data feeds based on at least one of a first modelidentifier of the first machine learning model or a second modelidentifier of the second machine learning model, wherein the first modelidentifier indicates a type of machine learning model of the firstmachine learning mode and the second model identifier indicates a typeof machine learning model of the second machine learning model.

9. The method of any of embodiments 7-8, wherein the updated applicationdata comprises a plurality of features, the method comprises: removingone or more features from the plurality of features to generate theproduction data.

10. The method of any one of embodiments 7-9, wherein the first datacomprises production data, and the production data is to be provided tothe plurality of machine learning models.

11. The method of any one of embodiments 1-10, wherein detecting thechangepoint comprises: determining that a value of a first feature ofthe first data differs from an expected value for the first feature bymore than a threshold amount.

12. The method of embodiment 11, wherein a BDML framework is used todetermine whether the first data includes one or more changepoints.

13. The method of any one of embodiments 1-12, further comprising:responsive to the changepoint being detected, causing a third machinelearning model to be executed on the first data to obtain second outputdatasets; and computing a second performance score for the third machinelearning model based on the second output datasets, wherein the secondmachine learning model is caused to execute on the first data based onthe first performance score and the second performance score satisfyingthe first condition.

14. The method of embodiments 13, wherein the first condition beingsatisfied comprises the first performance score and the secondperformance score being less than a threshold performance score.

15. The method of any one of embodiments 1-12 and 14, furthercomprising: determining that the first data is to be provided to thefirst machine learning model and a third machine learning model; andresponsive to the changepoint not being detected, preventing the thirdmachine learning model from being executed on the first data.

16. The method of any one of embodiments 12-15, wherein the firstmachine learning model and the third machine learning model have a firstexecution frequency; and the second machine learning model has a secondexecution frequency, the second execution frequency being less frequentthan the first execution frequency.

17. The method of any one of embodiments 1-12, 14, and 16, furthercomprising: determining that the first data is to be provided to a thirdmachine learning model having an execution frequency less than that ofthe first machine learning model; and prior to the third machinelearning model executing on the first data, preventing the third machinelearning model from executing on the first data based on the firstperformance score satisfying the first condition.

18. The method of any one of embodiments 1-12, 14, 16, and 18, whereinsecond output data is obtained based on the second machine learningmodel executing on the first data, the method further comprises:determining that the first data is to be provided to a third machinelearning model having an execution frequency less than that of the firstmachine learning model; and causing the third machine learning model tobe executed on the first data to obtain third output data, wherein thesecond machine learning model is assigned as a primary model and thethird machine learning model is assigned as a secondary model.

19. The method of embodiment 18, further comprising: computing a set ofperformance metrics for the second machine learning model and the thirdmachine learning model; and selecting the third machine learning modelto be assigned as the primary model and the second machine learningmodel to be assigned as the secondary model based on the set ofperformance metrics computed.

20. One or more tangible, non-transitory, machine-readable media storinginstructions that, when executed by one or more processors, effectuationoperations comprising those of any of embodiments 1-19.

21. A system comprising: one or more processors; and memory storingcomputer program instructions that, when executed by the one or moreprocessors, cause the one or more processors to effectuate operationscomprising those of any of embodiments 1-19.

What is claimed is:
 1. A system for optimizing resource allocation in amulti-thread, multi-dimensional machine learning environment, the systemcomprising: obtaining, via a data feed, production data to be providedto a plurality of machine learning models; detecting a changepoint inthe production data based on a value of a first feature of theproduction data being determined to differ from an expected value forthe first feature by more than a threshold amount; causing a firstmachine learning model to be executed on the production data to obtainfirst output datasets; computing, based on the first output datasets, afirst performance score for the first machine learning model; andcausing, based on the changepoint being detected in the production dataand the first performance score satisfying a first condition, anadditional machine learning model to execute on data from the data feed,wherein the first condition being satisfied comprises the firstperformance score being less than a threshold performance score.
 2. Amethod implemented by one or more processors configured to executecomputer program instructions, the method comprising: obtaining, via adata feed, first data to be provided to a plurality of machine learningmodels; detecting a changepoint in the first data; causing a firstmachine learning model to be executed on the first data to obtain firstoutput datasets; computing, based on the first output datasets, a firstperformance score for the first machine learning model; and causing,based on the changepoint being detected and the first performance scoresatisfying a first condition, an additional machine learning model toexecute on data from the data feed.
 3. The method of claim 2, whereinthe first condition being satisfied comprises the first performancescore being less than a threshold performance score.
 4. The method ofclaim 2, further comprising: building the additional machine learningmodel based on the changepoint being detected and the first performancescore satisfying the first condition.
 5. The method of claim 2, whereinthe first machine learning model has a first execution frequency, andthe additional machine learning model has a second execution frequencythat is less than the first execution frequency.
 6. The method of claim2, wherein obtaining the first data comprises: obtaining the first datavia a given data feed configured to receive updated application datafrom one or more real-time applications.
 7. The method of claim 2,wherein detecting the changepoint comprises: determining that a value ofa first feature of the first data differs from an expected value for thefirst feature by more than a threshold amount.
 8. The method of claim 2,further comprising: responsive to the changepoint being detected,causing a second machine learning model to be executed on the first datato obtain second output datasets; and computing a second performancescore for the second machine learning model based on the second outputdatasets, wherein the additional machine learning model is caused toexecute on the data from the data feed based on the first performancescore and the second performance score satisfying the first condition,wherein the first condition being satisfied comprises the firstperformance score and the second performance score being less than athreshold performance score.
 9. The method of claim 2, furthercomprising: determining that the first data is to be provided to asecond machine learning model having an execution frequency less thanthat of the first machine learning model; and prior to the secondmachine learning model executing on the first data, preventing thesecond machine learning model from executing on the first data based onthe first performance score satisfying the first condition.
 10. Themethod of claim 2, wherein additional output data is obtained based onthe additional machine learning model executing on the first data, themethod further comprising: determining that the first data is to beprovided to a further machine learning model having an executionfrequency less than that of the first machine learning model; andcausing the further machine learning model to be executed on the firstdata to obtain further output data, wherein the additional machinelearning model is assigned as a primary model and the further machinelearning model is assigned as a secondary model.
 11. The method of claim10, further comprising: computing a set of performance metrics for theadditional machine learning model and the further machine learningmodel; and selecting the further machine learning model to be reassignedas the primary model and the additional machine learning model to bereassigned as the secondary model based on the set of performancemetrics.
 12. A non-transitory computer-readable media storinginstructions that, when executed by one or more processors, causeoperations comprising: obtaining, via a data feed, first data to beprovided to a plurality of machine learning models; detecting achangepoint in the first data; causing a first machine learning model tobe executed on the first data to obtain first output datasets;computing, based on the first output datasets, a first performance scorefor the first machine learning model; and causing, based on thechangepoint being detected and the first performance score satisfying afirst condition, an additional machine learning model to execute on datafrom the data feed.
 13. The non-transitory computer-readable media ofclaim 12, wherein the first condition being satisfied comprises thefirst performance score being less than a threshold performance score.14. The non-transitory computer-readable media of claim 12, furthercomprising: building the additional machine learning model based on thechangepoint being detected and the first performance score satisfyingthe first condition.
 15. The non-transitory computer-readable media ofclaim 12, wherein the first machine learning model has a first executionfrequency, and the additional machine learning model has a secondexecution frequency that is less than the first execution frequency. 16.The non-transitory computer-readable media of claim 12, whereinobtaining the first data comprises: obtaining the first data via a givendata feed configured to receive updated application data from one ormore real-time applications.
 17. The non-transitory computer-readablemedia of claim 12, wherein detecting the changepoint comprises:determining that a value of a first feature of the first data differsfrom an expected value for the first feature by more than a thresholdamount.
 18. The non-transitory computer-readable media of claim 12,further comprising: responsive to the changepoint being detected,causing a second machine learning model to be executed on the first datato obtain second output datasets; and computing a second performancescore for the second machine learning model based on the second outputdatasets, wherein the additional machine learning model is caused toexecute on the data from the data feed based on the first performancescore and the second performance score satisfying the first condition,wherein the first condition being satisfied comprises the firstperformance score and the second performance score being less than athreshold performance score.
 19. The non-transitory computer-readablemedia of claim 12, further comprising: determining that the first datais to be provided to a second machine learning model having an executionfrequency less than that of the first machine learning model; and priorto the second machine learning model executing on the first data,preventing the second machine learning model from executing on the firstdata based on the first performance score satisfying the firstcondition.
 20. The non-transitory computer-readable media of claim 12,wherein additional output data is obtained based on the additionalmachine learning model executing on the first data, the operationsfurther comprise: determining that the first data is to be provided to afurther machine learning model having an execution frequency less thanthat of the first machine learning model; and causing the furthermachine learning model to be executed on the first data to obtainfurther output data, wherein the additional machine learning model isassigned as a primary model and the further machine learning model isassigned as a secondary model.